{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "description": "VO Simple Cone Search basic tutorial."
   },
   "source": [
    "# Searching major star catalogs and Simbad by position or name with `vo_conesearch`\n",
    "\n",
    "## Authors\n",
    "P. L. Lim\n",
    "\n",
    "## Learning Goals\n",
    "* Perform a cone search around M31 using a web service.\n",
    "* Write the result out to a LaTeX table.\n",
    "* Perform a SIMBAD query using the cone search result.\n",
    "* Extract metadata from the cone search catalog.\n",
    "* Sort cone search results by angular distance.\n",
    "* Search multiple cone search services at once (synchronously and asynchronously).\n",
    "* Estimate the run time of a cone search.\n",
    "\n",
    "## Keywords\n",
    "astroquery, table, coordinates, units, vo_conesearch, LaTex, SIMBAD, matplotlib\n",
    "\n",
    "## Summary\n",
    "This tutorial desmonstrates the [Cone Search](http://astroquery.readthedocs.io/en/latest/vo_conesearch/vo_conesearch.html) subpackage, which allows you to query a catalog of astronomical sources and obtain those that lie within a cone of a given radius around the given position. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python standard library\n",
    "import time\n",
    "import warnings\n",
    "\n",
    "# Third-party software\n",
    "import numpy as np\n",
    "\n",
    "# Astropy\n",
    "from astropy import coordinates as coord\n",
    "from astropy import units as u\n",
    "from astropy.table import Table\n",
    "\n",
    "# Astroquery. This tutorial requires 0.3.5 or greater.\n",
    "import astroquery\n",
    "from astroquery.simbad import Simbad\n",
    "from astroquery.vo_conesearch import conf, conesearch, vos_catalog\n",
    "\n",
    "# Set up matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are running an older version of `astroquery`, you might need to set `vos_baseurl` yourself, as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from astropy.utils import minversion\n",
    "\n",
    "if not minversion(astroquery, '0.3.10'):\n",
    "    conf.vos_baseurl = 'https://astroconda.org/aux/vo_databases/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, it might be useful to list the available Cone Search catalogs first. By default, catalogs that pass nightly validation are included. Validation is hosted by Space Telescope Science Institute (STScI)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conesearch.list_catalogs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's pick an astronomical object of interest. For example, M31."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = coord.SkyCoord.from_name('M31', frame='icrs')\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, a basic Cone Search goes through the list of catalogs and *stops* at the first one that returns non-empty VO table. Let's search for objects within 0.1 degree around M31. You will see a lot of warnings that were generated by VO table parser but ignored by Cone Search service validator. VO compliance enforced by Cone Search providers is beyond the control of `astroquery.vo_conesearch` package.\n",
    "\n",
    "The result is an [Astropy table](http://astropy.readthedocs.io/en/stable/table/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = conesearch.conesearch(c, 0.1 * u.degree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('First non-empty table returned by', result.url)\n",
    "print('Number of rows is', len(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This table can be manipulated like any other Astropy table; e.g., re-write the table into LaTeX format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.write('my_result.tex', format='ascii.latex', overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now use your favorite text editor to open the `my_result.tex` file, but here, we are going to read it back into another Astropy table.\n",
    "\n",
    "Note that the extra `data_start=4` option is necessary due to the non-roundtripping nature of LaTeX reader/writer (see [astropy issue 5205](https://github.com/astropy/astropy/issues/5205))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_tex = Table.read('my_result.tex', format='ascii.latex', data_start=4)\n",
    "print(result_tex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cone Search results can also be used in conjuction with other types of queries.\n",
    "For example, you can query SIMBAD for the first entry in your result above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Due to the unpredictability of external services,\n",
    "# The first successful query result (above) might differ\n",
    "# from run to run.\n",
    "#\n",
    "# CHANGE THESE VALUES to the appropriate RA and DEC\n",
    "# column names you see above, if necessary.\n",
    "# These are for http://gsss.stsci.edu/webservices/vo/ConeSearch.aspx?CAT=GSC23&\n",
    "ra_colname = 'ra'\n",
    "dec_colname = 'dec'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Don't run this cell if column names above are invalid.\n",
    "if ra_colname in result.colnames and dec_colname in result.colnames:\n",
    "    row = result[0]\n",
    "    simbad_obj = coord.SkyCoord(ra=row[ra_colname]*u.deg, dec=row[dec_colname]*u.deg)\n",
    "    print('Searching SIMBAD for\\n{}\\n'.format(simbad_obj))\n",
    "    simbad_result = Simbad.query_region(simbad_obj, radius=5*u.arcsec)\n",
    "    print(simbad_result)\n",
    "else:\n",
    "    print('{} or {} not in search results. Choose from: {}'.format(\n",
    "        ra_colname, dec_colname, ' '.join(result.colnames)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now back to Cone Search... You can extract metadata of this Cone Search catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_db = vos_catalog.get_remote_catalog_db(conf.conesearch_dbname)\n",
    "my_cat = my_db.get_catalog_by_url(result.url + '&')\n",
    "print(my_cat.dumps())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a favorite catalog in mind, you can also perform Cone Search only on that catalog. A list of available catalogs can be obtained by calling `conesearch.list_catalogs()`, as mentioned above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = conesearch.conesearch(\n",
    "    c, 0.1 * u.degree, catalog_db='The USNO-A2.0 Catalogue (Monet+ 1998) 1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Number of rows is', len(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's explore the 3 rows of astronomical objects found within 0.1 degree of M31 in the given catalog and sort them by increasing distance. For this example, the VO table has several columns that might include:\n",
    "\n",
    "* `_r` = Angular distance (in degrees) between object and M31\n",
    "* `USNO-A2.0` = Catalog ID of the object\n",
    "* `RAJ2000` = Right ascension of the object (epoch=J2000)\n",
    "* `DEJ2000` = Declination of the object (epoch=J2000)\n",
    "\n",
    "Note that column names, meanings, order, etc. might vary from catalog to catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_names = result.colnames\n",
    "print(col_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before sort\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After sort\n",
    "result.sort('_r')\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also convert the distance to arcseconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['_r'].to(u.arcsec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you want *all* the results from *all* the catalogs? And you also want to suppress all the VO table warnings and informational messages?\n",
    "\n",
    "__Warning: This can be time and resource intensive.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    all_results = conesearch.search_all(c, 0.1 * u.degree, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for url, tab in all_results.items():\n",
    "    print(url, 'returned', len(tab), 'rows')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick out the first one with \"I/220\" in it.\n",
    "i220keys = [k for k in all_results if 'I/220' in k]\n",
    "my_favorite_result = all_results[i220keys[0]]\n",
    "print(my_favorite_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Asynchronous Searches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Asynchronous versions (i.e., search will run in the background) of `conesearch()` and `search_all()` are also available. Result can be obtained using the asynchronous instance's `get()` method that returns the result upon completion or after a given `timeout` value in seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async_search = conesearch.AsyncConeSearch(\n",
    "    c, 0.1 * u.degree, catalog_db='The USNO-A2.0 Catalogue (Monet+ 1998) 1')\n",
    "print('Am I running?', async_search.running())\n",
    "\n",
    "time.sleep(3)\n",
    "print('After 3 seconds. Am I done?', async_search.done())\n",
    "print()\n",
    "\n",
    "result = async_search.get(timeout=30)\n",
    "print('Number of rows returned is', len(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async_search_all = conesearch.AsyncSearchAll(c, 0.1 * u.degree)\n",
    "print('Am I running?', async_search_all.running())\n",
    "print('Am I done?', async_search_all.done())\n",
    "print()\n",
    "\n",
    "all_results = async_search_all.get(timeout=30)\n",
    "for url, tab in all_results.items():\n",
    "    print(url, 'returned', len(tab), 'rows')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimating the Search Time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's predict the run time of performing Cone Search on `http://gsss.stsci.edu/webservices/vo/ConeSearch.aspx?CAT=GSC23&` with a radius of 0.1 degrees. For now, the prediction assumes a very simple linear model, which might or might not reflect the actual trend.\n",
    "\n",
    "This might take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    t_est, n_est = conesearch.predict_search(\n",
    "        'http://gsss.stsci.edu/webservices/vo/ConeSearch.aspx?CAT=GSC23&',\n",
    "        c, 0.1 * u.degree, verbose=False, plot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Predicted run time is', t_est, 'seconds')\n",
    "print('Predicted number of rows is', n_est)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the actual run time and number of rows to compare with the prediction above. This might take a while.\n",
    "\n",
    "As you will see, the prediction is not spot on, but it's not too shabby (at least, not when we tried it!). Note that both predicted and actual run time results also depend on network latency and responsiveness of the service provider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_real, tab = conesearch.conesearch_timer(\n",
    "    c, 0.1 * u.degree,\n",
    "    catalog_db='http://gsss.stsci.edu/webservices/vo/ConeSearch.aspx?CAT=GSC23&',\n",
    "    verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Actual run time is', t_real, 'seconds')\n",
    "print('Actual number of rows is', len(tab))"
   ]
  }
 ],
 "metadata": {
  "astropy-tutorials": {
   "author": "P. L. Lim",
   "date": "March 2015",
   "description": "...",
   "link_name": "VO Conesearch",
   "name": "",
   "published": false
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
